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AB STB ACT 

A study conducted by the School District of 
Philadelphia and that city's Federal Reserve Bank, entitled "«jat 
Works in Beading," was used as a case study to determine what factors 
influence the translation of research findings into educational 
policy. The Philadelphia study, which examined variables associated 
with achievement growth in reading, received extensive nationwide 
media attention and was used by the Philadelphia school 
superintendent's office in the development of recommendations for 
changes in reading instruction. Analysis of the study's design and of 
the methods used in disseminating its findings resulted in the 
identification of ten factors associated with its being translated 
into educational policy: (1) identification of the clients affected 
by the study's findings, (2) vitality of the topic, (3) participant 
involvement, (4) technical quality, (5) reporting formats, (6) 
findings keyed to the decision process, (7) preparation of policy 
makers, (B) overcoming resistance, (9) the role of the ombudsman, 
(10) the role of the entrepreneur. Factors associated with research 
utilization in urban areas were also identified. (Author/BE) 
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INTRODUCTION 



Urban school district-based research and evaluation offices have 
grown rapidly, almost uncontrollably, since the advent of the Elemen- 
tary and Secondary Education Act of 1965. The view that evaluation, as 
the process of providing information for decision makers, is vital to 
effective school system administration has developed simultaneously 
with increasing Federal involvement in education, particularly in urban 
areas. Although the overall recognition and expansion of the roles of 
educational research and evaluation should be viewed positively, such 
rapid development does not take place without complications. 

Offices of research and evaluation should function as service agencies 
that aid decision-making and advance instructional practices in school 
districts. The provision of information, whether of a contextual, forma- 
tive, or summative nature, presupposes that the information will be at 
least potentially useful. Its usefulness may depend on (1) the congru :e 
of the information and the expectations of those for whom it is being 
developed (the clients); (2) the way in which it is communicated to 
various clients; and (3) the ability of those clients holding decision 
authority to base educational policy upon it. 

That these factors are related is obvious. They may, in fact, be sim- 
plistically thought of as a paradigm in which the translation of research 
and evaluation findings varies according to the congruence bs^v^n in- 
formation and client expectations, and the communication of evaluation 
data. Though a useful framework, such a paradigm is much too skeletal 
to provide for an in-depth analysis of the factors that ultimately lead to 
the development of policy based upon research and evaluation findings. 

The focus of this paper will be upon the identification and explora- 
tion of certain subliminal factors, and the interrelationships between 
them, which may ultimately account for the translation of research and 
evaluation findings into policy. 1 will attempt to isolate and examine 
those factors that relate to the translation of research and evaluation into 
educational policy. Though a theoretical framework may serve as a 
hypothetical structure, the problem will be examined through a case 
study. The forces related to the acceptance and utilization of information 
will be viewed in the World of practice, not theory. 

Three major sections follow this brief overview of the paper s struc- 
ture. In the first, an actual case will be presented as a subject for analysis. 
The case study deals with a major piece of cooperative research com- 
pleted in May of 1979 by the .School District of Philadelphia and that 
city's Federal Reserve Bank. The study, entitled What Works in 
Reading? was conducted over a three-and-one-half-year period, but 
sprang from earlier works dating back to 1971 . The study's historical and 



political background will be discussed, as will the earlier root studies and 
their relationship to the present research. A general synopsis of the study 
and its key findings will be presented. Of particular importance to the 
focus of this paper, however, is the study's impp.^ ind its degree of ac- 
ceptance, both nationally and locally. It is thL cveptance that relates 
directly to the translation of the study's findings and suggestions into 
school district recommendations and policy decisions. ^ 
The section dealing with Analysis will attempt to explore why this 
particular itudy had great impact, while other, earlier attempts, did not. 
Planning for change, overcoming resistance, and building upon utiliza- 
tion factors will be viewed as part of an overall approach to educational 
development. Finally, factors leading to policy decisions will be explored 
in the concluding section. 



WHAT WORKS IN READING?- A C ASK STUDY 



Background 

The results of What Works in Reading? (a joint School District/ 
Federal Reserve Bank empirical study in Philadelphia) were released in 
June of 1979. Though controversial, the study has sparked an enormous 
amount of national interest and attention. The purpose of this study was 
to determine what makes a difference in teaching children to read. Years 
of experience of many thousands of teachers have testified to the fact 
that no one, simple, magical solution could be expected. This study, 
therefore, attempted id identify the factors that contributed most 
strongly to achievement growth in reading. In particular, investigators 
were interested in appraising the relative merits of approaches to reading. 

The study had its genesis in two independent efforts by the School 
District of Philadelphia and the Federal Reserve Bank of Philadelphia. 
As part of its ongoing research and evaluation activities, the Office of 
Research and Evaluation of the School District of Philadelphia had con- 
ducted several studies of reading practices in recent years. One study— of 
a single school that had demonstrated remarkable progress in reading 
during the course of one school year— was a post hoc attempt to describe 
the reading effort in as much detail as possible. It was largely a "fishing 
expedition" (Scheib and Comerford, 1972). The results were interesting, 
but served primarily as an impetus for a more intensive look at successful 
reading practices. 

A more extensive study, involving fifteen "successful" schools, was 
conducted in 1974. Th : s study involved interviews of administrators and 
students, as well as th' examination of a number of school characteris- 



tics. Though over eighty variables measuring these characteristics were 
examined, the results were again inconclusive. Variables that differen- 
tiated the schools studied from the other «less successful" schools in the 
system could not be identified. 

At the same time, the Federal Reserve Bank of Philadelphia was 
conducting a study of its own. In 1975, it published a study of achieve- 
ment growth in the Philadelphia schools, which utilized econometric 
techniques (Summers and Wolfe, 1975). This effort was reviewed exten- 
sively, especially by economists, but was criticized from within the 
School District of Philadelphia. Objections were raised on both technical 
and social grounds. After a period of heated interaction, it became 
evident that the School District had no intention of utilizing this study 
for policy development or decision-making purposes. 

In late October of 1975, the Superintendent of Schools of 
Philadelphia and the President of the Federal Reserve Bank of 
Philadelphia were invited by the Deputy Mayor of Philadelphia to meet 
with him to discuss the recently released Federal Reserve Study. Both 
brought members of their staffs to the meeting. The discussion that 
morning focused on what, if any, followup to the findings and 
recommendations of the Federal Reserve Bank report was being planned. 
A considerable amount of promise in the methodology and implications 
for longer range policy decisions was discerned in spite of School District 
comments critical of certain of the technical procedures of the report. 

An agreement was reached whereby the Office of Research and 
Evaluation of the School District and the Research Department of the 
Federal Reserve Bank would meet to discuss the development of a fol- 
lowup study utilizing, as a base, the methodology employed by the 
Federal Reserve Bank, but integrating within it the concerns and learning 
priorities of the School District. Shortly after this meeting at City Ha.l, 
the Execu ive Director of the Office of Research and Evaluation and the 
Research Officer and Economist of the Federal Reserve Bank met to 
discuss Ihe possibilities for a new study. 

Key staff members from both the Federal Reserve Bank and the 
School District met in a series of half-day work sessions to plan the 
study. At the outset, the tone was cordial but guarded, and the meetings 
were negotiating sessions as much as planning meetings. 

Reasons for earlier disagreement quickly became evident. The nature 
of various school variables had to be clarified. In addition, differences in 
statistical terminology were a barrier. At one point, a staff memoer 
familiar with both multiple regression analysis and analysis of variance 
was brought in to "translate" the terms and concepts used. 

As the work sessions progressed, they brought with them a 
heightened awareness of each other's world. Eventually, a genuine 
mutual respect developed. Given what had occurred in the past, this 
understanding and. regard were sine qua non for the implementation of 
this study. 
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Overview of the Study 

Some schools are more successful than others in terms of reading 
achievement growth from one year to the next, and in terms of annual 
level of achievement. Why does one school have a higher score than 
another? Why do some students perform better than others? By col- 
lecting measures of possible reasons, one might be able to discriminate 
among reasons by virtue of their degree of association with the measure 
of success. In short, it is possible to compute the multiple regression 
between the achievement measure (the dependent variable) and the esti-. 
mated measures of the hypothesized reasons (the predictors or indepen- 
dent variables). 

This study was a search for those variables that are associated with 
changes in achievement growth, and an attempt to describe the associa- 
tions in terms of socioeconomic conditions, educational inputs, and peer 
group characteristics. The cliche "correlation does not mean causation" 
is an appropriate caveat. However, in the world of educational policy, 
decisions must be made on the basis of the best available information. 
Therefore, policy implications were the end product of this study. 

Schools were picked for the study by rank-ordering all elementary 
schools by their overall average reading scores (1975, grades 1-4) and 
their overall gain scores (1974-1975, grades 1-4) in reading. Ten schools 
at the top of both lists and ten at the bottom 6f both ljsts were selected. 
Five schools »n the middle of both lists completed the sampleVf 25 
schools. 

Though pupils in this study were from the 25 selected schools, the 
study involved facts about individual pupils. It was the individual pupil 
who was the subject of this study. All of the fourth grade pupils in the 
schopls selected were in the study. Fourth grade was chosen because it is 
the grade in which a large drop in test scores is typically observed; it is the 
highest grade which is .found in all elementary schools; and the ability to 
read at that grade is importantly related to a student's subsequent 
progress. 

Information was gathered for the study by teams made up of the staff 
of the Office of Research and Evaluation and of the Division of English/ 
Reading Language Arts Education. These teams visited the schools in- 
volved and interviewed the principal, the teachers, the reading teachers, 
and the reading aides. A total of 25 principals. 25 reading teachers, 94 
classroom teachers, and 68 reading aides were interviewed. In addition, 
team members gathered large amounts of information* from each pupil's 
records. 

All of this information * 's checked and double-checked for accuracy 
and then placed on a comjutt; file. One hundred sixty-two variables 
about each of the 1,828 pupils made up the computer file on which the 
study Was based. Later, various combinations of these separate items of 
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information increased the total number of variables about each pupil to 

245. ' 
Five kinds of variables about each pupil were gathered: 

1. Variables about the principal of the pupil's school. 

2. Variables about the reading teacher in the pupil's school.. 

3. Variables about the pupil's classroom teacher. * 

4. Variables about the pupil's school. 

5. Variables about the individual pupil. 

Once all of the information about each pupil was placed on computer 
tape, a random sample of 25 percent of the pupils was removed from the 
file and put aside. This was kept as" a cross-validation sample, whici 
allowed the results to be double-checked. 

Over 500 multiple regression equations were run on the large (75 
percent) file. This invo>ed trying out many different combinations of 
facts (equations) about The pupils in order to determine which combina- 
tion was most closely related to gains in reading scores. After the 
equation that seemeu to be most closely related to gains in reading scores 
was found, it was used on the cross-validation (25 percent) file. Findings 
that were the same in both cases could be reported with some confidence; 
those that were not, could be presented for discussion and further study. 



Key Findings 

The rationale for the study was based in part on the notion that 
adding to the existing body of knowledge concerning successful practices 
in reading was only partially useful. The wealth of research in reading 
has resulted in the layering of both old and new ideas upon one another. 
Little attention has been paid to removing the irrelevant or unproductive. 
Since such excess baggage is often detrimental to instruction and costly in 
terms of resources, What Works in finding? attempted to identify 
k, variables that both did and did not ma V- a difference in reading achieve- 
ment. 

Variables That Made a Difference. 

1. Some things about ihe-puptt seemed to make a difference: 
-»-The better the pupil's attendance, the more the reading score 

increased. 

• The fewer unexcused absences, the more the reading score in- 
creased. 

• Pupils who attended kindergarten seemed to gain more than 
those who did not. 

2. - Some things about the principal seemed to make a difference: 

• Where principals had experience in the field , of reading, the 
pupils achieved more. 

• The more often the principal observed in the classroom, the more 
the pupils achieved. 

5 ' 8 , 
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4 3. Some things about the school seemed to make a * vfference: 

• When schools had more professional supportiv* staff per pupil, 
the pupils achieved more. 

• Pupils clearly achieved least in K-6 schools, most in K-7 schools, 
and 'n between in K-8 and K-5 schools. 

4. Somethings about the classroom teacher seemed to make a. 
difference: 

• Where teachers had fewest absences, the pupils achieved more. 

• 'Where teachers had more than one year expeiience teaching 
fourth grade, pupils achieved more. 

5. Some things about reading instruction seemed to make a 
difference: 

' • Pupils taught by the linguistic basal approach achieved distinc- 
tively more than pupils using other reading approaches. 

• The more minutes a week of sustained silent reading, the more 
pupils achieved. 

• The fewer reading aides per week, the more pupils achieved. 

• Where teachers felt that, if given a choice, they would pick the 
same reading program again, pupils achieved more. 

• Pupils in larger classes (up to 35) gained more than those in 
smaller classes (as few as 24). 

• Pupils in classes taught in small group/whole class combination 
achieved more than pupils who were taught individually only, in 
small groups only, or as a whole class only. 



Variables That Made No Difference. Though a number of things 
were found to make a difference, most did not. Certain of those varia- 
bles that did not make a difference were emphasized because, in the past, 
they had been considered important. It was found that the following 
things did not affect pupil achievement: 

1. The number of times a pupi*. had moved since starting school. 

2. Beii^ bused to relieve overcrowding. 

3. The amount of administrative experience the principal had. 

4. Whether or not the principal had a doctorate. 

5. The number of years the principal had been in the school. 

6. Whether the pupil came from a low or high-income neighbor- 
hood.' . 

7. Whether the scliool had more or fewer pupils from Jow-income 

, families. 

8. The race of the teacher. 

9. The number of years of experience of the teacher. 

10. The. number of graduate courses in reading and language arts 
taken by the teacher. ' % 

11. The number of minutes a day of reading instruction. 

12. The time of day "reading was taught. 
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13. The professional preparation and experience of the school read- 
ing teacher. ... . 
Policy Implications. The following actions were recommended based 

on the findings of this study: 

1. Establish an outreach program to assure that as many children 
as possible have a kindergarten experience. 

2. Explore the expansion of the use of the linguistic basal reading 

approach. , . 

' 3. Consider the establishment of a special task force to explore pro- 
grams for, and develop intervention studies of, the lowest 
achieving pupils. 

4. Develop or modify programs or policies to minimize pupil ana 
teacher absenteeism. 

5. Develop or modify programs or policies to increase the amount 
of teacjier time that is spent in direct instructional contact with 
pupils. 

6. Develop or modify progr?jns or policies to increase the amount 
of time principals spend observing in a classroom. 

7. Explore the development of programs to increase reading 
knowledge and experience for potential principals prior to their 
appointments. 

• 8. Explore reasons why the K-7 organization seemed to be asso- 
ciated with the greatest amount of pupil gain, and K-6 the least. 
9.' Consider mandating a policy of a daily sustained silent reading 

period in all schools. . 
10 Initiate discussions with appropriate governmental otticiais 
directed toward the modification of current regulations to permit 
the allocation of Federal resources on a pupil specific basis. 



The Study's Impact 

The very day the study was presented at a public Board of Education 
meeting in June of 1979, it began making news. Though purely of local 
interest at first, it was the subject of a wire service story featured in the 
Los Angeles Times the following week. By mid-summer, over a thousand 
inquiries had been received. 

Local television and radio stations broadcast stones on the results ot 
the study, with both of the all-news radio stations producing feature re- 
ports and four others taping brief interviews with the major authors. The 
three major daily metropolitan newspapers published extensive stories 
each focusing on a somewhat different aspect of the study, but all 

covering the major findings. «f th* 

The Inquirer emphasized the ' 'common pedag6gical wisdom of the 

study's results in its lead: 

What seemed obvious in the days of the little red school house has 
been confirmed by results of a major reading study released 
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yesterday by the Federal Reserve Bank and the Philadelphia School 
District. Basically , the study 'shows that children learn to read more 
easily in relation to the quality of time thty spend with their teachers 
(June 26, 1979). 

The Bulletin also focused upon the cont&cubetwcen students, teach- 
ers, and principals: 

The overriding principle of the study— a deceptively simple one, 
according to its authors— is that the most effective reading teachers 
are those who maintain the closest and most continuous contact with 
their pupils and school principal (June 25, 1979). J 

However,, it picked up on the term "short links' • utilized in the Board 
presentation: 

"You need direct, short links— that's the payoff,'* said Anita A. 
' Summers, a former Federal Reserve Bank economist and now a 
professor at the University of Pennsylvania's Wharton School. 
41 When you introduce extra links, you're reducing effectiveness with 
diversionary tactics." ' % 

The Daily News accentuated the cooperative nature df the study, and 
in doing so, anticipated a key utilization factor: 

Three years ago, when the Fed .issued a study that suggested im- 
provements to aid the learning of city school students, the Board of 
Education ignored the study because it was considered to be the work m 
of outsiders (June 25, 1979). * 
Following the initial local media coverage and the wire story filed by 
the Associated Press, a number of major 3 educational periodicals 
reported the study's results. Education USA, the most widely circulated 
weekly education newsletter, devoted three of its eight pages to it. The 
lead article, "New Study Finds What Worked in Reading/' focused 
upon the more potentially controversial findings and emphasized both 
what made no difference as well as what worked. A second article of the 
"analysis" variety was also published. 

The Education USA articles created an enormous demand for the 
report. By the end of the summer, over three-thousand inquiries had been 
received. The "Newsfront" section of the first Phi Delta Kappan of the 
new school year (September 1979) featured the stucfy in an article entitled 
"What's the Best Way to Teach Reading? Here's a Surprise." As a 
result, two thousand additional requests were received within the next 
eight weeks. 

Perhaps the major impact of the study, however, was the appearance 
of a document entitled A Blueprint for Academic Achievements which 
was erroneously viewed by many as a companion to the report. Twp 
months prior to its release, the Superintendent anticipated the Blueprint 
in comments reported by Education USA: 

Philadelphia Schools Supt. Michael P. Marcase praised the resear- 
chers for their "significant findings," which, he said, would result in 
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"specific recommendations* * to the Board of Education for major 
changes in reading instruction. . (July 2, 1979, v26, n43:8). 



A Blueprint for Academic Achievement 

The Blueprint, as the document came to be known, was a 22-page 
draft document transmitted by the Superintendent of Schools to his 
Executive Cabinet for review, discussion, and refinement. The Superin- 
tendent's charge to his Cabinet was to "forge a blueprint for action, 
that couldbe presented to the Board of Education by December 1979. 

The Blueprint was developed largely as a result of a charge to the 
School District's Office of Curriculum and Instruction to study "the 
problems of raising achievement, particularly in schools with greatest 
need." Unbeknownst io the authors of What Works in Reading? the 
report of the Curriculum Office was integrated with the research study 
. by other members of the Superintendent's staff. The result was an unsus- 
pected though pleasant surprise to the researchers, for the Blueprint had, 
indeed, drawn heavily upon their-work. In his introductory memoran- 
dum, the Superintendent stated: 

. I have read in-depth the research report, What Works in 
Reading? I believe it can prove to be one of the most significant 
reports on reading achievement in the recent history of public educa- 
tion, and I have included- several of its recommend, .ions in this 
document (1979:3). 
Actually, ten of the Blueprint's 36 specific recommended actions 
flowed directly from What Works in Reading? with an equal number at 
least tangentially related to the study. The text of the Bluepnnt often 
mentioned the study directly. For example, the section of the Blueprint 
dealing with the importance of the principal's leadership in the school 
states: 

Nowhere was this proven more graphically than in the research 
-report What Works in Reading? which reported conclusive, 
statistical evidence that significantly more reading achievement took 
place in schools where the principal took a direct, active part in put- 
ting together the reading .program and where tin principal spent a 
good portion of his or her time in the classroom monitoring and 
motivating reading' instruction (1979:6). 

. In the section devoted to system-wide standards in curricular areas, 

the Blueprint refers to: 

... one of the most powerful findings of the recent research, What 
Works in R-ading? was the superiority of the basal linguistic ap- 
proach to reading instruction. This method produced far more 
achievement growth than did any other form of reading instruction 
m the fourth grade in the Philadelphia public schools (1979:12). 
And in discussing the need for direct instruction: 
What Works in Reading? also found that the kind of classroom 
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organization that produced significantly greater achievement growth 
than any other was an organization wh*ue the teacher taught the 
whole group pan of the time, then btoke the class into clusters for 
small group instruction. 

Still another major finding was the benefit of silent reading which, 
according to the research, produced significant reading achievement 
' growth (1979:15). 

The specific actions recommended by the Blueprint for Academic 
Achievement directly traceable to the findings and policy implications of 
What Works in Reading? included the following: 

1. The Office of Staff and Leadership Development, with the help of the 
Office of Field Operations and the Office of Curriculum and Instruction, 
will recommend to the Superintendent of Schools, by December 1, 1979, 
procedures to establish an executive institute for principals, focusing ex- 
clusively on leadership development in the instruction ot basic skills 
(1979:7). 

2. Principals will reserve the time during the instructional day to visit class- 
rooms and other school areas where student activity is taking place 
(1979:9). ) . 

3. Wherever possible, central office and district offiife meeti- js involvtng 
principals are to be scheduled when students are not in attendance at 
school (1979:9). 

4. The Office of Curriculum and Instruction will report, in detail, to the 
Superintendent of Schools its evaluation of the basalMinguistic approach 
to reading instruction, together with its recommendation why this ap- 
proach . should, or should not, become the School ©fetrict of 
Philadelphia's basic approach to the teaching of reading. This recommen- 
dation will be made by December 1, 1979 (1979:12). 

5. Office of Curriculum and Instruction establish several teams of super- 
visors with expertise in improving basic skills instruction using inter-disci- 
plinafy approach. Teams will establish consistent on-going relationship 

* * with low achieving schools and initiate detailed plan for staff development 

emphasizing basic skills instruction in all curricular areas (1979:14). 

6. Principal will use local and Central Office resources to continually 
emphasize the importance of direct instruction. This emphasi%*ill be im- 
plemented by the principal's frequent visits to observe classrooil activities 
(1979:15). 

7. Aides will be used less in instructional roles and more*in non-instructional 
areas like roll taking and other classroom management responsibilities 
(1979:1*).- , * . 

8. The Office of Curriculum and Instruction will report to the Superintend- 
ent of Schools, by December 1, 1979; its evaluation of the whole 
group/small group classroom instruction process and why it should, or 

• should not, become a standard form of reading instruction in our elemen- 
tary schools (1979:15). 

9. Thd Office of Cftrricuhim and Instruction will acv#*lop instruc- 
tional processes in the basic skills, particularly in reading, utilizing 
significantly more silent reading than is now the custom (1979:16). 

10. The Task Force on* Absenteeism will report to the Superintendent of 
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Schools, by December I, 1979, its recommendations for significantly 
decreasing teacher absenteeism (1979:16). 



ANALYSIS 



As demonstrated in the previous section, What Works in Reading? 
was not just another \>\cct of research. Unlike many studies that are 
politely accepted and then shelved to gather dust, this study has already 
had an impact. The extent cf that impact and the uhimate level of utiliza- 
tion remain to be seen. It is evident, however, that What Works in 
Reading? has succeeded as a catalyst for change where other such efforts 
have fanod. Why" . L - 

There are a number of possible answers to that question, with no 
single explanation alone accounting for the difference. My own partici- 
pation in the study and in tne process of its explanation and implementa- 
tion lead me to suggest f our possible reasons for the study's impact: (1) it 
maximized outside involvement while establishing internal ownership, 
(2) it took a proactive approach to planning for change, (3) it attempted 
to minimize resistance to change, and (4) it attempted to build upon fac- 
tors associated with research utilization. 

Ownership 

From the outset, great care was taken to establish the study as one 
undertaken by the School District, with assistance, from the Federal 
Reserve Bank, not the converse, as had been the case in the previous 
study Which School Resources Help Learning? In *?ite of its potential 
utility, the previous study had virtually no local impact because it was 
viewed as the work *»f "outsiders,." unfamiliar with both education and- 
the school system. The problems encountered in that first study were 
aptly summed up in an Inquirer feature on the role of the 4 outsider 
economist: 

That report was attacked from all sides. She was an outsider, her . 
study uninvited. So. as she puts it. "the institutional wall went 
down." Her suggestions were ignored. As a result of that report. said 
Michael Kean. research director for the city schools and a critic of 
the 1975 study, "a good deal happened, but not in Philadelphia. 
That experience was not forgotten, however: 

"What I learned," Ms. Summers says, "was when you come as a 
total outsider to the system, the wall is . . . nearly impenetrable. Peo- 
ple coalesce against the outsider" (June 30, 1979). 
In order to avoid the problem, Kean and Summers carefully planned 
the nature of the cooperative effort that was to ensue. Not only was the 
study's ownership to be associated with the School District, but the 
District's Office of Research and Evaluation took great care to involve 
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other key individuals and organizational divisions as well. 

From the initial conceptualization of the study, the Office #f Curricu- 
lum and Instruction, particularly its Division of Reading/English Lan- 
guage Arts, were heavily inyolved. The Associate Superintendent for 
Curriculum and Instruction and the Executive Director for Reading/ 
English Language Arts actively participated in the formulation of the 
problem and the preliminary study design. As the work progressed, the 
Division of Reading/English Language Arts became even more inv olved, 
with its Executive Director and several of her key assistants playing ma- 
jor roles in identifying the critical variables to be studied, designing the 
instrumentation, and helping to define the parameters of the effort. 

The actual data collection procedures were cooperatively handled by 
.supervisory staff members of both the Office of Research and Evalua 
tion and the Division of Reading/English Language Arts. The nature of 
such a joint endeavor amazed the former "outsider," according to the 
Inquirer: 

Mrs. Summers was amazed by the ease of gathering data this time. 
For her earlier study, with economist Barbara -Wolfe (now at the 
University of Wisconsin), of children in 150 schools, Ms. Summers 
"went up and down, knocking on school doo«-s, asking for the data 
and getting stonewalled all along the way. That data came from 
sweat, lots of sweat." It took her a year and a half to get the facts. 

This time, it took the school district three weeks. "I almost 
fainted," said Ms. Summcs (June 30, 1979). 
A final collaborative effort took place shortly before the results were 
publicly released. The Associate Superintendent for Curriculum and In- 
struction and the Executive Director of Reading/English Language Arts 
and her staff were provided with draft copies of the report and asked to 
review them. Their comments were shared during the course of two face- 
to-face meetings, and in virtually all instances their suggestions were in- 
tegrated within the text of the final document. 

Hence, the ownership of the study could be truly ascribed to both the 
School District's Office of Research and Evaluation and the office 
responsible for the programmatic area (reading) being examined. 



Proactive Planning 

The approach discussed in the preceding section is exemplary of the 
proactive approach to planning associated with the study. "Proactive" is 
used in contrast to "reactive." A common misconception is that plan- 
ning, by its very nature, must be proactive. The fact is, however, that a 
considerable amount of planning 'is undertaken in response to stimuli 
rather than for catalytic purposes. The proactive approach to planning 
anticipates the reaction to a situation and develops a desirable positive 
alternative response to it. The principal architects of the What Works in 
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Reading? study attempted to employ a proactive approach to planning 
from its outset. 

A variety of potential problems were identified during the course of 
the authors' initial meetings, and each was dealt with as if the success of 
the entire study depended upon its resolution. A host of problems were 
identified. Areas ranging from finances, scheduling, and personnel allo- 
cation, to political reaction, protocol, and technical verification were all 
considered. In the end, this attention to detail seemed to produce hand- 
some results. 

Among the financial problems confronted were the lack of separate 
funding and the' need to secure in-kind contributions of resources. 
Scheduling became a particularly critical problem, especially within the 
arenas *of data processing support and school-site data collection ac- 
tivities. The majority of the potential political and protocol-related pro- 
blems were dealt with in a manner similar to that described in the 
preceding section. Involvement and constant progress updates proved 
the key to such concerns. 

Perhaps most crucial was the attention paid to the planning of the 
release of the study's tesults. Here the "outside" partner in the 
study— the Federal Reserve Bank— was used as a neutral party in con- 
vening a press briefing prior to the formal public presentation. The 
Superintendent of Schools had, of course, already been briefed, and the 
study had been placed on the agenda of the meeting of the Board of 
Education to be held the following Monday. 

The education writers of the city's major nev spapers and all-news 
radio stations were invited to a luncheon at the Federal Reserve Bank; so 
were members of the newspapers' editorial boards. The study was 
presented carefully and non-technically, with nc limit on the time taken 
to answer questions. Each person present had agreed in advance to a 
"gag rule" until after the study was released publicly. Copies of the 
study were distributed, and all participants departed from the briefing 
far more knowledgeable than they would have been had they merely 
heard What Works in Reading? reviewed at a public Board of Education 
meeting. Each member of the fourth estate also had three days to review 
the document and to prepare a story for "immediate" release the follow- 
ing week. 

The results were gratifying, both in terms of coverage and accuracy. 
A particular dividend was the editorial attention paid to the study, and 
the suppoti provided by the editorial writers. For example, the Bulletin 
noted: 

Johnny's— and Jane's— reading difficulty is a perennial question 
batted around by educators and parents. The "what works" study, 
therefore, ought to be required reading in educational circles In our 
region and Washington. It might change some outlooks, and funding 
patterns, for the better (July 6, 1979). 
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The Inquirer even acknowledged the special briefing: 

They conducted a rather low-key briefing at the bank several days 
before their report was formally presented to the Board of Education 
and made public. What Works in Reading? is the title of the report 
but its findings have potential impact on school budgets as well as on 
•Hssroom techniques (July 13, 1979). 

Before cautiously endorsing the concept and the potential follow-up: 
This study is not the last word, of course, on why Johnny can or 
can't read, but it should stimulate new discussion about educational 
methods. It also should spur critical examination of old concepts 
about how to spend the school tax dollar (July 13, 1979). 

The hoped-for results were, indeed, achieved. 



Minimizing Resistance 

Resistance to change was identified in a previous section of this paper 
as a force to be contended with in planning for the translation of research 
and evaluation results with educational policy initiatives. The ownership, 
involvement, and communications strategies already described in this 
section doubtlessly helped serve to reduce some of the expected resis- 
tance. Two other related approaches may have also had a positive effect. 

Almost immediately following the formal public release of the study, 
its principal authors began making presentations to a large number and 
variety of audiences. Almost every group requesting a presentation has 
been serviced, and though some of the audiences have been antagonistic 
to selected results, the vast majority have appreciated the researchers 
"take it to the people" approach. "It's as if you really want us to under- 
stand the study," one participant commented. "1 thought research was 
supposed to be more technical." 

In addition to the "traveling road show" dissemination strategy, in- 
depth central office discussions were initiated. The Superintendent of 
Schools had promised at the time of the study's release to appoint a select 
committee to consider follow-up action. Close coordination between the 
Superintendent's Office and the Oifice of Research and Evaluation 
resulted in the appointment to the Committee of a broadly representative 
group of individuals. Organizations such as The Home and School 
Association, the Council of District Superintendents, Association of 
School Administrators, Federation of Teachers, as well as key program- 
mati' and supportive service personnel, formed the nucleus of the com- 
mittee. Though the group has moved slowly in considering the myriad 
options before it, its work has been both deliberate and productive. 

The authors of the study attempted to be just as deliberate in their at- 
tempt to minimize the technical condescension and egocentric prescrip- 
tion that so often permeate major studies. Wherever possible, the 
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authors cautioned their readers that the study was not a panacea and that 
, they must bear in mind: 

• This study was conducted only on fourth graders. It is not likely 
that fourth graders are completely different from all other grades, 
but one should be careful in thinking that all of the findings would be 
exactly the same for first graders or twelfth graders, for example. 

• Educational programs in schools are ever changing. This study was 
conducted during the fifth year of the District Reading Programs. 
Some of the things found to be of no importance at that point might 
possibly have been of critical importance during the first year or two 
of the program. 

• This study suggested what combination of things made a difference 
in reading achievement. It did not tell why. To try to find out why, 
additional studies of a different kind would be needed. 

• As conditions change, as the reading program changes, and as sug- 
gestions from this study are tried out in the schools, it would be im- 

-ponsnno repeat this study in order to find out something about the 

impact of these changes on reading achievement. 

The authors indicated that the findings presented were the ones that 
seemed most pertinent, and on which it was possible for the school 
system to take action. The findings were the result of inten? ; ve investiga- 
tion using sophisticated mathematical techniques. As such, it was sug- 
gested that they should certainly be given serious consideration and 
thought. 

There is, however, no guarantee that implementation of recommen- 
dations based on these findings would cause any miracles. There are 
too many things involved to make any such guarantee. These find- 

. tags are, however-, the best information available at this time. Even 

without guarantees, it is better to be guided by such information than 
to try to make decisions in a vacuum (Kean 1979:12). 

Not only did the researchers add qualifiers and disclaimers where 
they deemed them necessary and appropriate, they even attempted to an- 
ticipate some of the key questions relative to the impact of certain 
methods or problems upon higher and lower achieving students. Antici- 
pation of the public's concern led the authors to include in the report, for 
example, that: , 

• The lowest achieving pupils did better if they were in higher 
. achieving schools. 

• The lipquistic basal reading approach was more beneficial 
to middle and higher achieving pupils than other reading 
approaches, but the reading approach utilized did not make 
any difference for the lowest achieving pupils. 

• Unexcused absences were equally harmful to higher and 
lower achieving pupils. ; 

• Having attended kindergarten was equally beneficial to 
higher and lower achieving pupils. 
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Utilization Factors 



One of the principal foci of this paper is the question, Why do certain 
research and evaluation reports have impact upon decisions while others 
do not? Brickell and Aslanian's recommendations for the communica- 
tion of research data, and those of Alkin et al. on factors affecting 
utilization, seem to apply to this case study. Though no attempt was 
made to tailor the .means of reporting the study's results to fit Brickeil 
and Aslanian's recommendations, the level of congruence is remarkable, 
as detailed in Figure 1 . 

Figure 1 

THE CONGRUENCE BETWEEN BRICKELL AND ASLANIAN'S 
RECOMMENDATIONS FOR THE COMMUNICATION OF RESEARCH DATA " 
AND THE REPORTING OF THE RESULTS OF WHAT WORKS IN READING? 



Brickell and AsktaiatTs 


Reporting the Resulis of What Works 


Recommendations 


in Reading? 


Brevity 


Eight page Summary utilizing bullets to 


highlight major points, plus a four- 




page Digest of Results listing the fin- 




dings according to each sector. 


Placing most technical 


Regression results and means and 


material in appendices 


standard deviations included in tables 




at end of the report, A separate — 




Technical Report developed to answer 




technical Questions and concerns. 


Timelines with respect 


Information released according to the 


to decision makers 


specific planning and decision-making 


expectations 


needs of ioo administration. 



Use of entirely non- Econometric and educationese 

technical language "translated" into common language," 

understandable by lay and professional 

personnel alike. 



Provision of public 
presentation material to 
amplify the executive 
summary 



Multiple copies of the study's summary, 
instead of large graphics, disseminated 
at the outset of each public 
presentation. 



The series of eight factors affecting utilization cited by Alkin et al. 
also bear a high degree of similarity to the concerns taken into considera- 
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tion by the researchers in planning, implementing, and disseminating the 
results of the study. The foregoing discussion of ownership, proactive 
planning, and minimizing resistance to change demonstrates the atten- 
tion paid to (1) setting pre-defined boundaries, (2) orienting information 
users, (3) the approach of the study team, (4) the study team's credibility, 
(5) organizational structure and relationships, (6) contextual/en- 
vironmental forces, (7) report content and style, and (8) the needs and ex- 
pectations of decision makers: y 

The Urban Factor 

Although each of the above factors is important, none is necessarily 
unique to planning within the urban educational milieu* The experience 
associated with the Philadelphia Study, however, was certainly unique in 
terms of both process and impact because it was nurtured and grew in an 
urban context. What, then, are the principles that malce educational 
planning in urban areas unique? I believe they fall into four broad areas: 
" Scale. The sheer magnitude of urban problems provides more fertile 
fields of inquiry than might normally be expected. In addition, there are 
greater availability of information and greater number of potential sub- 
jects. Typically, cities are studied with greater frequency and, as a result, 
larger quantities of data are often readily available. This is also true of 
the variety of problems associated with urban living and the number of 
agencies responsible for maintaining information about urban dwellers. 

Large urban areas tend, also, to be associated with greater flexibility 
in identifying innovative appr oache s. Urban school systems are more 
likely to secure meaningful support from business groupsi civic groups or 
local foundations, all of which may have more direct stakes- in the 
schools than school districts where resources and encouragement are 
more typically a notary Club luncheon or an American Legion achieve- 
ment award. 

Resources. Offices of research and evaluation are more prevalent in 
larger urban school districts than elsewhere, although recent research has 
shown that such offices are spreading rapidly. Urban areas typically con- 
tain more specialists with the training and experience necessary to staff 
such offices. In addition, the size of the school district and the resultant 
reporting requirements often necessitates the maintenance of such an of- 
fice. Without an existing organizational structure and an individual 
designated as responsible, large-scale research projects simply cannot be 
carried out. 

Greater and more specialized non-school resources also tend to be 
available in urban areas. In the case of the Philadelphia Study, the 
School District's partner was the Federal Reserve Bank. Large univer- 
sities, businesses, and non-profit organizations with interests in educa- 
tion are more commonly located in cities than elsewhere. 

Greater variety of dissemination possibilities exist in cities than 
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elsewhere. Local electronic media are critically related to the ultimate im- 
pact of studies like the one described in this paper. Though non-urban, 
areas have newspapers and local radio stations, the mass coverage ac- 
corded by multiple local TV stations and all-news radio stations, not tc 
mention large-circulation daily newspapers and neighborhood weeklies, 

plays a pivotal role. . ee 

Complexity. The complexity of the urban scene most certainly affects 
the way in which educational planning takes place. The heterogeneity ot 
the constituencies served is perhaps the major consideration. Such 
heterogeneity often creates widely varying expectancies and needs As a 
result, differing solutions are necessary. The greater number of solu- 
tions, the more complex the systems for developing the solutions. Hence 
there is a greater demand for information and planning. 

The political diversity of urban issues adds to the complexity. Diver- 
sity means there is less chance for the achievement of a ready consensus. 
Therefore, the market for studies of the type described in this paper may 
be better in urban environments. 

Economics. America's poverty population is more heavily concen- 
trated in urban areas than elsewhere. The needs of urban dwellers are 
numerically greater and tend to create great strain upon the cities limited 
resources. This may create a demand for greater efficiency and produc- 
tivity, which, in turn, may produce a ready market for potential adop- 
tion and utilization of research a.id evaluation findings 

In urban areas, school sys»em finance is frequently controlled, at 
least in- pan, by agencies other than school boards. Very often urban 
schools are funded partially by the city government. As a result, urban 
school districts may be fiscally less self-determined and may have a 
greater need for information concerning successful practices. In addi- 
tion the way the school district spends its money will be more closely 
scrutinized by other agencies and by the local population. 

Finally, urban areas reflect an economically diverse constituency. 
Diversity is the key, for it creates varied expectations in terms of the role 
of the public school system and the resources associated with it. 

CONCLUSIONS 

Examination of this topic, with its descriptive case study and 
analysis, has generated ten over-all factors associated with the transla- 
tion of research and evaluation into educational policy. These ten factors 
certainly do not comprise a finite listing; they relate to the case studied 
here, but may not necessarily apply to all similar situations Perhaps even 
more important, this listing may well represent only a fraction of the 
total range of such factors. The ten general areas are as follows: 

1 Identification of Clients. It is important to single out the specific 
clienis and client groups who are most closely associated with the 
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research being undertaken and most directly affected by any policy deci- 
sions likely to be developed as a result of that research. The expectations 
of the clients, as well as the type of information and services they require, 
are important considerations in focusing the research and evaluation 

process. . 

2. Vitality of the Topic. There are a great many topics whose thrust is 
so tangential to the solution of a particular problem that they will seldom 
generate very much interest. A topic directly related to a highly visible 
problem may virtually insure that the results of the study of that topic 
will be carefully considered. In framing a topic for study, criteria such as 
"importance, visibility, acceptability, and understandability should be 
reviewed. 

3. Participant Involvement. This factor relates directly to the 
previous discussion of ownership. It is important to involve as many as 
possible of those groups who will likely be ultimately affected by the 
study. Involvement should be genuine, not merely "window dressing." 
Though this may lengthen the period of the study, it will pay handsome 
dividends in the long run. 

4. Technical Quality. The technical aspects of the study (in terms of 
research methodology, appropriate measurement techniques, and careful 
reporting) must be beyond reproach. Wherever possible, the study 
should impiove upon previous major works in the area. In addition, the 
use of multiple measures will greatly enhance the study's acceptability. 

5. Reporting Formats. A variety of reporting "formats is very 
necessary if a study's results are to be adequately communicated. Since 
most decisions are made by laymen, the basic report should be as 
nontechnical as possible and should always include an executive sum- 
mary and/or abstract. In addition, an illustrated, popularized version 
tends to be very useful in communicating with the public at large, as is 
the use of multimedia materials such as slides, large graphics, etc. A full 
technical report should be available for that small group of individuals 
interested in the technical specifics of the study. 

6. Findings Keyed to the Decision Process. Timing is all-important if 
research and evaluation results are to have genuine impact upon policy. 
In order to perfect such timing, it may be useful to attempt to determine 
the type of decisions likely to be made as a result of the study. Wherever 
possible, a study's results and recommendations for policy should relate 
to the general context of the school system and should be demonstrative- 
ly cost effective and/or cost efficient. 

7. Preparation of Policy Makers. Advance briefings for key decision 
and policy makers are of tremendous value. Briefings should include a 
nontechnical overview of the specific results of the study, the relation- 
ship of those results to the needs of the system, and the specific implica- 
tions for policy that seem to spring from the study. Those studies actual- 
ly "commissioned" by decision makers probably stand the greatest 
chance of ultimately having an impact. 



8. Overcoming Resistance. The ability of the study team to anticipate 
potential problems resulting from their work will help reduce resistance. 
The degree to which resistance can be dealt with prior to its surfacing 
will, in the long run, minimize disruption of both the implementation of 
the study and the ttanslation of its results into practice. 

9. 'The Role of the Ombudsman. The designation of a member of the 
study team as ombudsman or trouble-shooter during the study's im- 
plementation will help reduce both resistance and misunderstanding. 
Such an individual should initiate dialog, not wait for problems to occur. 
The ombudsman should attempt to work with clients prior to, during, 
and after implementation of the study. 

10. The Role of the Entrepreneur. Finally, the entrepreneurial role is 
an important correlate of policy decisions. This role may be played by 
either the individual(s) resporisiblKfor producing the research and/or 
evaluative information or the decision maker ultimately responsible for 
translating it into policy and acting upon it. If the policy maker exhibits 
entrepreneurial behavior, the researcher's role is considerably simplified. 
Since this is not typically tb* case, however, it is the potential impact of 
the researcher's entrepreneurial acumen that will be considered here. 

Within the discipline of economic development, the entreprene j is 
an individual who applies a new combination of resources and 
technology in productive activity to effect change. The re sulting chang e^ 
. ___howeverr-4s-rarely only-economic.- Ir is- social as well, ror economic 
change does not occur without social ramifications. There is a great deal 
of similarity between the roles and characteristics of the entrepreneur 
and the change agent. By either classification, such an individual has 
been the key figure in originally unlocking the doors of development in 
many of what are thought of today as the world's progressive nations. 
The entrepreneurial role is not only appropriate, but extremely useful in 
working toward the acceptance of research information and ultimately 
translating it into policy. In playing the role of the entrepreneur, the 
researcher cannot assume a neutral posture. He must, in fact, be a strong 
advocate of the utilization of the information he has developed and 
should approach his task with an eye toward its translation into policy 
from the outset. It is important to remember that advocacy of the utiliza- 
tion of data need not compromise objectivity in the conduct of research 
or evaluation. 

Because he anticipates future possibilities, the entrepreneur becomes 
adept in long-range planning: McClelland has suggested that "The suc- 
cessful entrepreneur. . .is by definition someone who considers. . .alter- 
natives and consequences before they actually happen to him" (Mc- 
Clelland 1961:237). Organizational skills also play an important role in 
the entrepreneur's success. He is generally equally skilled as an ad- 
ministrator and as an expert in public relations. 

In conclusion, it has been said that "imitation is the highest form of 
flattery." Ixi the case of research utilization, however, that saying might 
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well be revised to "institutionalization is the strongest indicator of suc- 
cess." 

The Blueprint for Academic Achieve nent actually served to institu- 
tionalize the' type of study that' created much of the impetus for the 
Blueprint's initial development. That document's final section deals with 
"Evaluation as a Tool for Achievement Growth" and emphasizes that: 

. . .it will be important foffthe Office of Research and Evaluation to 
conduct follow-up studies which examine the effectiveness of various 
changes made to improve school achievement. These studies should 
look at specific recommendations and compare variolis overall 
strategies implemented in schools. This suggests that proposed 
changes should be implemented in a systematic way so that the effec- 
tiveness of different strategies can be measured and compared. 
In the final analysis, however, institutional acceptance is not entirely 
sufficient. It is institutional demand for decision information thai is 
perhaps the single most vital factor in the translation of research and 
evaluation into educational policy. 
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